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Description 



on a measurement point from two different directions, 
and a 3-dimensional position and an orientation of aface 
(plane) in the vicinity of the measurement point calcu- 
lated from the measurement result. 
[001 0] On the other hand, in a case where such a pat- 
tern light is a spol light, measurement may be made by 
projecting spot lights on Ihree or more different points 
which are not on a straight line,and a 3-dimentional po- 
sition and an orientation of a plane on which these points 
acStyVn^ite p"osftio'nte' stored in a robot * o are positioned calculated from the measurement result. 

[0011] In this case, three points positioned near one 
another may be sclectcd.and one of thoso points sot as 
a target measurement point, or three points (or more 
points) positioned near a target measurement point may 
15 be selected and a 3-dimensional position and an orien- 
tation of a plane on which the target measurement point 
is positioned obtained from the measurement results of 
these points. For example, the measurement position 
can be caused to correspond to the geometrical center 



[0001 ] The present invention relates to a robot teach- 
ing apparatus used in an operation for teaching a motion 
position to an industrial robot. 

[0002] As a typical technique which is often employed 
for leaching a motion position of a robot., there is a so- 
called playback system. When this technique is used, a 
robot is manually moved to a position where the robot 
must move 
controller. 

[0003] As another typical technique, there is a so- 
called off-line teaching. In this technique, using an off- 
line programming device, an environmental model in- 
cluding a robot and/or an object to be worked is estab- 
lished and a position to which the robot has to move is 
obtained according to the model by a calculation The 
teaching position thus obtained is stored in a robot con- 
troller, for example, through a transfer. „ , „,„ 
r00041 When the playback system is employed, since 20 of gravity position formed by the three points, 
positions required for an operation of the robot (posi- *' - ' »'< f J ' ' ^ ' 



tions to be taught) are taught to the robot one by one, 
much time and labor are required. It is not rare that the 
number of teaching positions required for an actual work 
amounts to a large numbex (f OLexampJe, ,100). 
[0005] On the other hand, when the off-line teaching 
is utilized, it is necessary to input a model of a motion 
object into an off-line teaching device. In this case, there 
is a drawback in that, unless a shape of the object is 
defined as a CAD model or the like in advance, it is dif- 
ficult to input the shape accurately and much time is re- 
quired for inputting the shape model. 
[0006] An object of the present invention Is to provide 
a robot position teaching apparatus which can perform 
an operation position teaching operation of a robot eas- 
ily with accuracy. Also, another object of the invention 
is to provide a robot position teaching apparatus which 
is suitable particularly for a movement path teaching for 
a robot performing operation while moving along a linear 
or curved operation route. 

[0007] In the present invention, positions of points on 
an operating line of a work object are measured by a 
3-dimensional position and attitude measuring device 
(sensor) constituted by a camera and light projecting 
means. Teaching point data for operating a robot is cre- 
ated from information about 3-dimensional positions of 
points constituting the working line, which is obtained 
from this measurement. 

[0008] The light projecting means projects a pattern 
light (which is typically a slit light or a spot light) in the 
vicinity of a point to be measured (hereinafter, referred 
to as a "measurement point"). An object projected with 
the pattern light is photographed by a camera, and in- 
formation about a 3-dimentional position of the portion 
on which the pattern light is projected is captured by im- 
age analysis. 

[0009] In a case where such a pattern light is a slit 
light, measurementmay be made by projecting slit lights 



[0012]^ Briefly speaking, in the present invention, a 
3-dimensional position sensor provided with light pro- 
jecting means of such a pattern light as a slit light, a spot 
light or the like, and a camera are utilized so that a 3-di- 
25 mensional position of a target measurement point is 
measured. Also, an inclination (orientation) of a surface-=-- 
(an area near the measurement point which can be re- 
garded as a flat plane) is calculated together with the 
measurement of the 3-dimensional position. The atti- 
30 tude of the target measurement point can be represent- 
ed by this orientation of the surface. 
[0013] Incidentally, one can use the 3-dimensional 
position sensor provided with light projecting means of 
such a pattern light as a slit light, a spot light or the like. 
35 and a camera, to obtain a 3-dimenlional position of a 
target measurement point. That is, by using data ob- 
tained through calibration regarding the light projecting 
means and the camera, a 3-dimensional position of a 
target point can be calculated within a sensor or the like 
40 from the position of a measurement point on an imago 
obtained by the camera and a light projecting state (data 
representing projecting orientation and position of the 
pattern light) at this time. 

[001 4] In a case where the light projector or the cam- 
45 era of the 3-dimensional position sensor is mounted on 
a distal end portion of a hand of a robot, the 3-dimen- 
sional position of the target point can once be obtained 
on a coordinate system (for example, a tool coordinate 
system) fixed on the distal end portion of the hand of the 
robot. Even in such a case, by using data about the cur- 
rent position of the robot at this time, the coordinate sys- 
tem can easily be converted to a coordinate system (for 
example, a world coordinate system, a work coordinate 
system or the like) fixed to a working space. 
[0015] Here, when the above target measurement 
points are sequentially defined and measured along a 
working line on an object to be measured, teaching 
points for a working robot for performing work along the 



2 



BNSUUCID. --EP _ ... IPISOI'/Ai! I - 



3 



EP 1 215 017 A2 



4 



working line can be obtained. 

[0016] In view of the above, a prefered embodiment 
of robot teaching apparatus according to the present in- 
vention is a robot teaching apparatus of a type wherein 
a teaching robot mounted with light projecting means for 
projecting a slit light or a spot light on an object as a 
pattern light and a camera is used to perform working 
along a working line on the object, and comprises: 
means forobtaining a 3-dimensional position and an ori- 
entation of a face of an object on which the pattern light 
has boon projected, from an imago of the camera about 
the face of the object; means for specifying points con- 
stituting the working line on the image of the camera; 
and means for obtaining the specified 3-dimentional po- 
sitions of the points constituting the working line, on the 
basis of the 3-dimensional position and the attitude of 
the face and the positions of the points on the camera 
image constituting the working line. 
[0017] By using a robot teaching apparatus according 
to the present invention, it becomes unnecessary to per- 
form a playback motion on an object robot to be taught. 
Also, it is not required to constitute a model for off-line 
teaching, so that a teaching operation for a robot can 
easily be performed. 

[0018] The above and other features of the present 
invention will beeome-apparent from explanations the 
following embodiments with reference to the attached 
drawings, in which: 

Fig. 1 is a perspective view showing a schematic 
arrangement in a first embodiment of the present 
invention; 

Fig. 2 is a perspective view showing a schematic 
arrangement In a second embodiment of the 
present invention; 

Fig. 3 is a perspective view showing a schematic 
arrangement in a third embodiment of the present 
invention; 

Fig. 4 is a perspective view showing a schematic 
arrangement in a fourth embodiment of the present 
invention; and 

Fig. 5 is a perspective view showing a schematic 
arrangement in a fifth embodiment of the present 
invention. 

[001 9] A first embodiment of the present invention will 
be explained with reference to Fig. 1 . 
[0020] Reference numeral 5 denotes a robot which is 
an object to be taught. Mounted at an arm distal end of 
the robot 5 are an image pickup device (camera) 2 and 
a projector 1 . The projector 1 has a function for project- 
ing a pattern light. In the embodiment shown in Fig. 1, 
a slit light projecting type one is employed as the pro- 
jector 1 . 

[0021] A robot controller 6 is of a type that an image 
processing device is housed therein, and it is connected 
to the robot 5, the projector 1 , the image pickup device 
2, a display device 3 and a teaching operation panel (not 



shown). Provided in this robot controller 6 are not only 
a hardware and a software for a normal robot control, 
but also a hardware and a software for instructing pro- 
jecting operations (on/off, a projecting direction, etc.) of 
5 the projector 1 , and a hardware and a software for per- 
forming photographing action or the image pickup de- 
vice 2 or taking-in of photographed images, performing 
required image processing, transmission of image sig- 
nals to the display device 3 or the like. 
'0 [0022] Incidentally, it is assumed here that a position 
teaching for performing working on an object to be 
worked has not been carried out yet. Also, it is assumed 
here that a reference work 7 is an object which serves 
as a reference for a teaching operation and it is posi- 
15 tioned correctly in a working space for teaching. In other 
words, it is assumed here that this reference work 7 is 
maintained at a position and with an attitude with which 
the robot 5 performs working actually at a reproduction 
operation time after a position teaching has been com- 
20 pieted. 

[0023] For teaching, first, the robot 5 is caused to 
move with a manual operation to a position and an atti- 
tude at which it is properly spaced from the reference 
work 7 and it looks over the entire working space. Next, 
2s an image including the reference woric7jna field of view 
is picked up and the image is displayed on the image ~ 
display device 3. 

[0024] A 2-dimensional image position pointing de- 
vice 4 utilizing a mouse or the like is provided in the im- 
30 age display device 3, and an operator points a position 
(measurement starting point) to be measured by the ro- 
bot 5 on the image using this pointing device 4. The 
pointed position (uO, vO) on the image is stored in the 
' robot controller 6. 
35 [0025] Such pointing of a position to be measured can 
include not only pointing of the measurement starting 
point in the manner described above but also include 
pointing of a plurality of points constituting a working line 
on an image by using the pointing device 4, specifying 
40 of a working line by moving the pointing device 4 along 
the working line, specifying of a window which specifies 
a region where a working line exists on an image by us- 
ing the pointing device 4, and so on. 
[0026] The measurement starting point is ordinarily 
4S selected on a working line on which an actual working 
is performed and in the vicinity of a work starting point. 
For example, in a case of a welding robot, the working 
line corresponds to a welding line and the work starting 
point corresponds to a starting end of the welding line. 
so [0027] After the measurement starting point is speci- 
fied, a view line (a 3-dimensional straight line connecting 
the measurement starting point and a center of a lens 
of the image pickup device 2) corresponding to the po- 
sition (uO, vO) is obtained using the position (uO, vO) on 
« the 2-dimensional image and the position and the orien - 
tation of the image pickup device 2. The robot is caused 
to approach to the reference work 7 such that a project- 
ing direction of the projector 1 does not deviate from the 
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view line, so that it is moved up to a position where a 
3-dimensional position on the reference work 7 and an 
inclination ot the face at the position can be measured. 
[0028] Next, a slit light is projected from the projector 
1 onto the reference work 7, measurement ot a 3-dimen- 
sional position on the reference work 7 and an inclina- 
tion (orientalion) of a face in Ihe vicinily Ihereol is start- 
ed. A projecting direction (a position and a direction of 
a bright line formed on the reference work 7 by a slit line) 
of a slit line at the measurement starting time is adjusted 
such that it crosses the working lino in the vicinity of the 
specified measurement starting point. An image includ- 
ing an image of the bright line formed on the reference 
work 7 is photographed by the camera 2 and a 3-dimen- 
sional position (x1 . y1 , z1) of the measurement starting 
point is measured according to the principle of a tngo- 
nometrical survey. The position on the image corre- 
sponding to this measurement starting point is defined 
as (u1 , v1 ). Since the principle of the measurement and 
processing performed for measurement by the robot 
controller 6 are well known, description thereof will be 
omitted here. 

[0029] The robot is moved along a working path a 
small distance by a small distance, and light projecting 
and photographing are performed for each small dis- 
tance movement, so that 3-dimensional positions (x2, 
y2, z2), (x3. y3, z3)... of points at which the bright line 
crosses the working line are sequentially obtained. 
[0030] The 3-dimentional positions (xi, yi, zi) [i = 1 , 
2...] of a sequence ot the points which constitute the 
working line thus obtained are utilized as teaching points 
of the robot to produce a movement path or a motion 
program of the robot. Also, orientations (al, bi, ci) TJ =1 . 
2...] of faces in the vicinity the working line are measured 
in correspondence lo the respective points. The number 
of faces to be measured is considered to be ordinarily 
two, but it may be one or three or more in some cases. 
In such a case where an attitude of a tool center point 
is taught to the robot, these face orientation data and 
unit vectors (di, ei, fi) P = 1 , 2...) representing the direc- 
tion of the working line can be utilized. 
[0031] Incidentally. *ere may be a case where a re- 
dundant point is thinned from a plurality of detected 
points, or a case where the sequence of the points de- 
tected is approximated to a straight line or an arc line by 
the method of least squares or the like to reproduce the 
teaching points of the robot. In such a case, the points 
detected do not correspond to the teaching points di- 
rectly. 

[0032] Also, there may be a case where the points 
constituting the working line can not be specified, by rea- 
son of erroneous detection of line from the image of the 
camera, failure of detection of the working Une, and so 
on Therefore . it is preferable to introduce a step for ver- 
ifying the detection results. Specifically, as shown in Fig. 
1, the image of the camera 2 is displayed on a screen 
of the image display device 3, and the detected point 
positions (a sequence of points 10) constituting the 



working line are displayed while being superimposed on 
this image. 

[0033] Alternatively, the detected working line is dis- 
played on image display device 3 with the color orthick- 
5 ness changed while being superimposed on the image 
or the camera 2 displayed on the image display device 3. 
[0034] An operator confirms the image and the delect- 
ed positions displayed on the image (when they can not 
be detected, such a fact is notified to the operator 
10 through a message). In a case where a correct position 
has not boon dotoctcd, tho correct position is specified 
on the imago using tho position pointing device 4 (a 
mouse of a computer, for example) on the 2-dimensional 
image. 

15 [0035] The 3-dimensional position of the detected po- 
sition is re-calculated from the position specified on the 
image and the plane measured. Thereby even when the 
point which has been detected from the image of the 
camera is erroneous or even when the point has not 
20 been detected, the teaching point data can be created 
according to the instruction of the operator. 
[0036] "Furthermore, one of typical methods for de- 
tecting a working line from a camera image is a tech- 
nique using a pattern matching. In this technique, a line 
25 shape to be detected on an image is stored as a teach- 
ing P attern1n^adTance;"aricl apposition of an image pat- 
tern similar lo the teaching pattern is detected from an 
input camera image at a time of detection. In this tech- 
nique, since a pattern similar to the teaching pattern is 
30 detected from the image, the detected portion extending 
along the working line must be similar to the teaching 
pattern. 

[0037] In an actual object to be worked, however, 
there may occur a case where It is difficult to detect a 
35 complete route using only one leaching pattern because 
the working line on the image does not appear uniformly 
due to a reflection state of a surface of the object, an 
illumination state around the object, or the like. In view 
of this fact, in the present embodiment, when a working 
40 line has been detected from an image correctly, an im- 
age of the working line detected is stored as a teaching 
pattern, and the next detection is performed using a 
teaching pattern created from the image thus detected. 
[0038] Thereby, since the teaching data is always up- 
45 dated on the basis of the previous detection result, even 
when the working line appears in a changing manner on 
the image, the working line can be detected without in- 
fluence of such a change of appearance as compared 
with a case where only one teaching data piece is used. 
50 Here, the method for recording a teaching pattern has 
been explained with the example of the pattern match- 
ing, but a method for detecting an object using feature 
amount on an image (for example, a direction, an inten- 
sity or the like of an edge point) is generally applicable 
55 to a working line detection. 

[0039] Further, when a line shape is detected, it is 
preferable to selectively use an optimal detecting meth- 
od on the basis of a difference in shape such as a 
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straight line, a curved line, an arc or the like, but it is 
difficult to always recognize the shape of a line from the 
image data correctly. Therefore, an operator indicates 
the shape of a line to be detected so that a more pref- 
erable detecting method can be applied to a working line 
detection. 

[0040] A second embodiment of the invention will be 
explained with reference to Fig. 2. 
[0041 ) M ounted at an arm distal end of a robot 5 which 
is an object to be taught are an image pickup device 
(camera) 12 and a projector 1. The projector 1 has a 
function for projecting a pattern light. In the embodiment 
shown in Fig. 1 , a slit light projecting type one is em- 
ployed as the projector 1 in the same manner as the 
case of the first embodiment (Fig. 1). However, in this 
embodiment, a fixed camera 8 for photographing a 2-di- 
mensional image, which is arranged so as to overlook 
a whole working space with a proper distance is utilized 
as an image pickup device (camera). 
[0042] A robot controller 6 is of a type that an image 
processing device is housed therein, and it is connected 
to the robot 5, the projector 1 , the fixed camera 8, a dis- 
play device 3 and a teaching operation panel (not 
shown). This robot controller 6 may be one similar to 
that in the first embodiment. A reference work 7 serves 
as a reference for a teaching work and it is positioned^ 
accurately in a teaching work space. 
[0043] For teaching, first, the robot 5 is caused to 
move, for example, with a manual operation to a position 
and an attitude suitable for projection with a proper dis- 
tance from the reference work 7. Next, an image includ- 
ing the reference work 7 in afield of view is picked up 
using the fixed camera 8 and the image Is displayed on 
the Image display device 3. A position (measurement 
starting point) to be measured by the robot 5 is pointed 
on an image using the image position pointing device 4. 
Coordinates (uO, V0) of the position pointed on the im- 
age are stored in the robot controller 6. Since another 
method for pointing a position to be measured is similar 
to that in the first embodiment, explanation thereof will 
be omitted. 

[0044] The measurement starting point is ordinarily 
selected on a working line on which an actual working 
is performed and in the vicinity of a work starting point. 
After the measurement starting point is specified, a view 
line (a 3-dimensional straight line connecting the meas- 
urement starting point and a center of a lens of the fixed 
camera B) corresponding to the position (uO, vO) is ob- 
tained using the position (uO, vO) on the 2-dimensional 
image and the position and the orientation of the fixed 
camera 8. The robot 5 is caused to approach to the ref- 
erence work 7 in a manner such that a projecting direc- 
tion of the projector 1 does not deviate from the view 
line, so that it is moved up to a position where a 3-di- 
mensional position on the reference work 7 and an in- 
clination of the face at the position can be measured. 
[0045] Next, a slit light is projected from the projector 
1 onto the reference work 7, measurement of a 3-dimen- 



sional position on the reference work 7 and an inclina- 
tion (orientation) of a face in the vicinity thereof is start- 
ed. For the measurement, a slit light is projected on the 
reference work 7 from the projector 1 . A projecting di- 
s rection of a slit line at the measurement starting tlme(a 
position and a direction of a bright line formed on the 
reference work 7 by a slit line) is adjusted such thai it 
crosses the working line in the vicinity of the specified 
measurement starting point. An image including an im- 
10 age of the bright line formed on the reference work 7 is 
photographed by the fixed camera 8 and a 3-dimonsion- 
al position <x1 , y1 , z1 ) of the measurement starting point 
is measured according to the principle of a trigonomet- 
rical survey. 

is [0046] Also, like the first embodiment (Fig. 1 ), such a 
technique can be employed in which an image including 
an image of the bright line is photographed by a camera 
1 2 at a distal end of a robot arm and the 3-dimensional 
position (x1. y1. z1) at the measurement starting point 
20 is measured according to the principle of the trigonomet- 
rical survey. 

[0047] The position on the image corresponding to 
this measurement starting point is defined as (u1 , v1). 
Since the principle of the measurement and processing 
_f 5 Pf rformed for measurement by the robot controller 6 are 
~ weirknbwn; "description thereof will be omitted here. 
[0048] The robot is moved along a working path a 
small distance by a small distance, and light projecting 
and photographing are performed for each small dis- 
30 tance movement, so that 3-dimensional positions (x2, 
y2, z2), (x3, y3, z3)... of points at which the bright line 
crosses the working line are sequentially obtained. 
[0049] The 3-dimentional positions (xi, yi, zi) [1 = 1, 
2...] of a sequence of the points which constitute the 
& working line thus obtained are utilized as teaching points 
of the robot to produce the movement path or a motion 
program of the robot 5. Also, orientations (ai, bi, ci) [i =1 , 
2...] of faces in the vicinity the working line are measured 
in correspondence to Hie respective points. The number 
4 o of faces to be measured are considered to be ordinarily 
two, but it may be one or three or more in some cases. 
In a case where an attitude of a tool center point is taught 
to the robot, these face orientation data and unit vectors 
(di, ef fi) [1=1, 2...] representing the direction of the 
« working line can be utilized. 

[0050] Besides the above, this embodiment is not so 
different from the first embodiment regarding a thinning 
process of redundant detected points, an approximating 
process using a straight line or an arc, a countermeas- 
» ure for an erroneous detection or a detection disenable 
state (re-designation of measurement points and meas- 
uring them), and an application of a pattern matching 
process orthe like, so that repetitive explanation will be 
omitted. 

ss [0051] A third embodiments the invention will be ex- 
plained with reference to Fig. 3. 
[0052] Mounted at an arm distal end of a robot 5 which 
is an object to be taught is a projector t. The projector 
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1 has a function for projecting a pattern light. In the em- 
bodiment shown in Fig. 3, a slit light projecting type one 
is employed as the projector 1 in the same manner as 
in the cases shown in Figs. 1 and 2. 
[00531 In this embodiment, a camera 8 for photo- 
graphing a 2-dimensional image is mounted at a dislal 
end of an arm of a robot 15 (referred to as a second 
robot) which is provided in addition to the robot 5 (re- 
ferred to as a first robot). The second robot 15 consti- 
tutes a moving device of a camera 8. The camera 8 
mounted on the second robot 1 5 is utilized for teaching 
tho first robot 5. 

[00541 A robot controller 6 is of a type that an image 
processing device is housed therein, and it is connected 
to the first robot 5, the second robot 1 5, the projector 1 , 
the camera 8, the display device 3 and the teaching op- 
eration panel (not shown). 

[00551 This robot controller 6 may be one similar to 
that in the first embodiment. However, a hardware and 
a software for controlling the second robot 15 is further 
provided in the robot controller 6. A reference work 7 
serves as a reference for a teaching work and it is po- 
sitioned accurately in a teaching work space. 
[0056] For teaching, first, the first robot 5 is caused to 
move.forexample.with amanufll^peration.to.aposition 
and an attitude suitable for projection with a proper dis- 
tance from the reference work 7. Similarly, the second 
robot 15 is caused to move to a position and an attitude 
where it can include the reference work 7 in its field of 
view with a proper distance from the reference work 7. 
[00571 Next, an image including the reference work 7 
in the field of view is captured using the camera 8 mount- 
ed on the second robot 15, and the Image is displayed 
on the image display device 3. A position (measurement 
starling point) to be measured by the first robot 5 is point- 
ed on an image using an image position pointing device 
4 (referto an arrow). Coordinates (uO, VO) of the position 
pointed on the image are stored in the robot controller 
6. Since another method for pointing a position to be 
measured is similar to those in the first and second em- 
bodiments, explanation thereof will be omitted. 
[0058] The measurement starting point is ordinarily 
selected on a working line on which an actual working 
is performed and in the vicinity of a work starting point. 
After the measurement starting point is specified, a view 
line (a 3-dimensional straight line connecting the meas- 
urement starting point and a center of a lens of the cam- 
era 8) corresponding to the position (uO, vO) is obtained 
using the position (uO, vO) on the 2-dimensional image 
andthe position and the orientation of the camera 8. The 
first robot 5 is caused to approach to the reference work 
7 such that a projecting direction of the projector 1 does 
not deviate from the view line, so that it is moved up to 
a position where a 3-dimensional position on the refer- 
ence work 7 and an inclination of the face at the position 
can be measured. 

[00591 Next, a slit light is projected from the projector 
1 onto the reference work 7, measurement of a 3-dimen- 



sional position on the reference work 7 and an inclina- 
tion (orientation) of a face in the vicinity thereof is start- 
ed A projecting direction (a position and a direction of 
a bright line formed on the reference work 7 by a slit 
5 light) of the slit light at the measurement starting time Is 
adjusted such that it crosses the working line in the vi- 
cinity of the specified measurement starting point. An 
image including an image of the bright line formed on 
the reference work 7 is photographed by the camera 8 
io (or by the camera 1 2 mounted on the distal end of the 
first robot 5 for teaching) and a 3-dimonsional position 
(x1 , y1 . 21 ) of tho measurement starting point is meas- 
ured according to the principle of a trigonometrical sur- 
vey The position on the image corresponding to this 
rs measurement starting point is defined as (u1 , v1 ). Since 
the principle of the measurement and processing per- 
formed for measurement by the robot controller 6 are 
well known, description thereof will be omitted here. 
[0060] The robot is moved along a working path a 
20 small distance by a small distance, and light projecting 
and photographing are performed for each small dis- 
tance movement, so that 3-dimensional positions (x2. 
y2, z2), (x3, y3, z3)... of points at which the bright line 
crosses the working line are sequentially obtained. 
25 [0061] The 3-dimentional positions (xi, yi, zi) [i = 1 , 
2 ] of a sequence of the points which constitute the 
working line thus obtained are utilized as teaching points 
of the robot to produce the movement path or a motion 
program of the robot 5. Also, orientations (ai. bi, ci) [i =1 . 
30 2 ] of faces in the vicinity of the working line are meas- 
ured in correspondence to the respective points. The 
number of faces to be rneasured is considered to be or- 
dinarily two, but It may be one or three or more In some 
cases. In such a case where an attitude of a tool tip end 
as point is taught to the robot 5. these face orientation data 
and unit vectors (di, ei, fi) p = 1 , 2...] representing the 
direction of the working line can be utilized. 
[0062] Besides the above, this embodiment is not so 
different from the first and second embodiments regard- 
40 ing a thinning process of a redundant detected points, 
an approximating process using a straight line or an arc, 
a countermeasure for an erroneous detection or a de- 
tection disenable state (re-designation of measurement 
points and measuring them), an application of a pattern 
43 matching process and the like, so that repetitive expla- 
nation will be omitted. 

[0063] A fourth embodiment of the invention will be 
explained with reference to Fig. 4. 
[0064] Mounted at an arm distal end of a robot 5 which 
so is an object to be taught are a projector 1 1 and a camera 
2. The projector 11 has a function for projecting a spot 
light as a pattern light within a predetermined range in 
any direction instructed from a robot controller 6. 
[00651 The robot controller 6 is of a type that an image 
55 processing device is housed therein, and it is connected 
to the robot 5. the projector 11 . the camera 2, a display 
device 3 and a teaching operation panel (not shown). 
[0066] This robot controller 6 may be one similar to 
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that in the first embodiment (Fig. 1). However, a hard- 
ware and a software are slightly changed in correspond- 
ence to change of the projector 1 1 to the spot light pro- 
jecting type. Such a change is well known and therefore 
explanation thereof will be omitted here. A reference 
work 7 serves as a reference for a leaching work and it 
is positioned accurately in a teaching work space. 
[0067] For teaching, first, the robot 5 is caused to 
move, for example, with a manual operation, to a posi- 
tion and an attitude where it can include the reference 
work 7, by the camera 2 in its field of view with a proper 
distance from the roforencc work 7. 
[0068] Next, an image including the reference work 7 
in the field of view is captured using the camera 2, and 
the image is displayed on the image display device 3. A 
position (measurement starting point) to be measured 
by the robot 5 is pointed on an image using a pointing 
device 4 provided on the image display device 3{refer 
to an arrow). Coordinates (uO. V0) of the position point- 
ed on the image are stored in the robot controller 6. 
Since another method for pointing a position to be meas- 
ured is similar to those in the above-mentioned embod- 
iments, explanation thereof will be omitted. 
[0069] The measurement starting point is ordinarily 
selected on a working line on which an actual working 
- is perforrmrj-ana-itri he-vicinity of a work starting point. 
Afterthe measurement starting point is specified, a view 
line (a 3-dimensional straight line connecting the meas- 
urement starting point and a center of a lens of the cam- 
era 2) corresponding to the position (u0, vO) is obtained 
using the position (u0. vO) on the 2-dimensional image 
and the position and the orientation of the camera 2. 
Then, the robot 5 Is caused to approach to the reference 
work 7 such that a projecting direction of the projector 

I does not deviate from the view line, so that it is moved 
up to a position where a 3-dimensional position on the 
reference work 7 and an inclination of the face at the 
position can be measured. 

[0070] Next, a spot light is projected from the projector 

II onto the reference work 7, measurement of a 3-di- 
mensional position on the reference work 7 and an in- 
clination (orientation) of a face in the vicinity thereof is 
started. A projecting direction (the position of a bright 
point formed on the reference work 7 by a spot light) at 
the measurement starting time is adjusted so as to be 
positioned in the vicinity of the specified measurement 
starting point. An image including an image of the bright 
point formed on the reference work 7 is photographed 
by the camera 2 and a 3-dimensional position (xt , y1 , 
z1) of the measurement starting point is measured ac- 
cording to the principle of a trigonometrical survey. The 
position on the image corresponding to this measure- 
ment starting point is defined as (u1 , v1 ). Since the prin- 
ciple of the measurement and processing performed for 
measurement by the robot controller 6 are well known, 
description thereof will be omitted here. 
[0071] Incidentally, for measuring an orientation of a 
face around the 3-dimensional position, measurement 



of at least three points is performed for detection of each 
face in one robot position. These three or more points 
are spaced from one another by a small distance in a 
manner such that they are not aligned on one straiqht 
s line. 

[0072] The robot is moved along a working path a 
small distance by a small distance, and light projecting 
and photographing are performed for each small dis- 
tance movement, so that 3-dimensional positions of the 
*0 bright points (a plurality of points for each step) are se- 
quentially obtained. 

[0073] Tho 3-dimensional position of the working line 
is obtained on the basis of respective positions of faces 
around the working line thus obtained, and points on the 
fS working line are utilized as teaching points to create a 
movement path of the robot. 

[0074] For example, in a case where the working line 
corresponds to a crossing line of two planes A and B, 
the crossing line of the planes A and B is calculated in 
so the robot controller 6, and sampling points at predeter- 
mined intervals on the crossing line are selected as 
teach ing points and the 3-dimensional positions of these 
teaching points are stored as positions of the teaching 
points. Also, it is possible to teach the attitude of the 
25 robot 5 (an attitude of a tool center pointj^orijfcej^s^^. 
of the face orientation data of the planes A and B and 
data of unit vectors representing the direction of the 
working line. 

[0075] Besides the above, this embodiment is not dif- 
30 ferent from the above embodiments regarding a thin ning 
process of a redundant detected points, an approximat- 
ing process using a straight line or an arc, a counter- 
measure for an erroneous detection or a detection dis- 
enable state (re-deslgnation of measurement points and 
35 measuring them), an application of a pattern matching 
process or the like, so that repetitive explanation will be 
omitted. 

[0076] A fifth embodiment of the invention will be ex- 
plained with reference to Fig. 5. 
40 [0077] A first robot 5 serves as a teaching robot, and 
a second robot 1 5 provided separately of this first robot 
5 is a working robot for performing an actual work. That 
is, the first robot 5 is used in teaching for movement 
along the taught positions and the second robot 15 per- 
45 forms an actual work such as a welding work. 

[0078] In this embodiment, an image pickup device 
(camera) 12 and a projector 1 are mounted at a arm dis- 
tal end portion of the first robot 5 in charge of the teach- 
ing work. The projector 1 has a function for projecting a 
5 " pattern light and a projector of a slit light projecting type 
is employed as the projector 1 . On the other hand, a 
working tool 9 is mounted at an arm distal end of the 
robot 15 in charge of an actual work. 
[0079] A robot controller 6 is of a type that an image 
55 processing device is housed therein, and ft is connected 
to the first robot 5, the second robot 15, the projector 1 , 
the image pickup device (camera) 12, a display device 
3 and a teaching operation panel (not shown). 
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[0080] This robot controller 6 may be one similar to 
that in the first embodiment (Fig. 1). However, a hard- 
ware and a software for controlling the second robot 1 5 
are provided in the robot controller 6. A reference work 
7 serves as a reference for a teaching work and It is po- 
sitioned accurately in a teaching work space. 
[0081] For leaching, Tirst, the hrst robot 5 is caused to 
move, for example, with a manual operation, to a posi- 
tion and an attitude suitable for projecting and photo- 
graphing with a proper distance from the reference work 

[0082] Next, an image including the roferonco work 7 
in the field of view is captured using the camera 1 2, and 
the image is displayed on the image display device 3. A 
position (measurement starting point) to be measured 
by the first robot 5 is pointed on an image using a point- 
ing device 4 provided to the image display device 3 (re- 
fer to an arrow). Coordinates (uO, V0) of the position 
pointed on the image are stored in the robot controller 
6 Since another method for pointing a position to be 
measured is similar to those in the above-mentioned 
embodiments, explanation thereof will be omitted. 
[0083] The measurement starting point is ordinanly 
selected on a working line on which an actual working 
— ^performed and in the vicinity of a work starting point. 
Afterthe measurement starting point is specified, a view 
line (a 3-dimensional straight line connecting the meas- 
urement starting point and a center of a lens of the cam- 
era 1 2) corresponding to the position (u0, vO) is obtained 
using the position (uO. vO) on the 2-dimensional image 
and the position and the orientation of the camera 12. 
Then, the robot 5 is caused to approach to the reference 
work 7 such that a projecting direction of the projector 
1 1 does not deviate from the view line, so that it is moved 
up to a position where a 3-dimensionat position on the 
reference work 7 and an inclination of the face at the 
position can be measured. 

[00841 Next, a slit light is projected from the projector 
1 onto the reference work 7, so that measurement of a 
3-dimensional position on the reference work 7 and an 
inclination (orientation) of a face in the vicinity thereof is 
started. A projecting direction (a position and a direction 
of a bright line formed on the reference work 7 by a slit 
light) of the slit light at the measurement starting time is 
adjusted such that it crosses the working line in the vi- 
cinity of the specified measurement starting point. An 
image including an image of the bright line formed on 
the reference work 7 is photographed by the camera 2 
and a 3-dimensional position (x1 , y1 , z1) of the meas- 
urement starting point is measured according to the 
principle of a trigonometrical survey. The position on the 
image corresponding to this measurement starting point 
is defined as (u1 , v1 ). Since the principle of the meas- 
urement and processing performedf or measurement by 
the robot controller 6 are well known, description thereof 
will be omitted here. 

[0085] The first robot 5 is moved along a working path 
a small distance by a small distance, and light projecting 



and photographing are performed for each small dis- 
tance movement, so that 3-dimensional positions (x2, 
y2, z2), (X3, y3, z3),... at which the bright line crosses 
the working line are sequentially obtained, 
s [0086] The 3-dimentional positions (xi, yi, zi) (1=1. 
2...) or a sequence of the poinls which constitute the 
working line thus obtained are utilised as leaching poinls 
of the robot to produce the movement path or a motion 
program of the second robot 15. Also, orientations (ai, 
io bi, ci) [i -1 , 2...] of faces in the vicinity the working line 
arc measured to correspond to the respective points. 
The numbor of faces to bo measured is considered to 
be ordinarily two. but it may be one or three or more in 
some cases. In such a case where an attitude of a tool 
15 center point is taught to the robot 5, these face orienta- 
tion data and unit vectors (di. ei, fi) [i = 1 .2...] represent- 
ing the direction of the working line can be utilised. 
[0087] Incidentally, in a case where the coordinate 
system which serves as a reference of motion of the sec- 
20 ond robot 1 5 for working is different from the coordinate 
system which serves as a reference of motion of the first 
robot 5 for teaching, conversion is performed between 
both the coordinate systems using a transformation 
equation so that the teaching data for the second robot 
25 15 can be obtained. Since how to obtain a transforma- 
tion equation betweelTSfifferent motroTTJbsse-coordinate 
systems between two robots (in general, expressed by 
a homogeneous transformation matrix of 4 row and 4 
column) is well-known, explanation thereof will be omit- 
30 ted. For example, when there is a world coordinate sys- 
tem common to both the coordinates, transformation 
can be performed through the world coordinate system. 
[0088] Besides the above, this embodiment is not dif- 
ferent from the above embodiments regarding a thinning 
35 process of a redundant delected points, an approximat- 
ing process using a straight line or an arc, a counter- 
measure for an erroneous detection or a detection dis- 
enable state (re-designation of measurement points and 
measuring them), an application of a pattern matching 
40 process or the like, so that repetitive explanation will be 
omitted. 

[0089] Also, in the embodiments of the present inven- 
tion shown in Figs. 1 to 5, the projector 1 or 11 and the 
2-dimensional image pickup device 2 or 12 which are 
45 mounted on the robot may be mounted on the robot for 
teaching at a time of teaching and they may be detached 
from the robot when the robot performs working. Since 
the measuring devices such as the projector or the im- 
age pickup device is detached from the robot at the time 
50 of working in this manner, a merit or an advantage that 
an interference between the projector or the 2-dimen- 
sional image pickup device and the object to be worked 
can be avoided can be achieved and also a set of the 
measuring devices can be used commonly for a plurality 
55 of robots for teaching. 

[0090] The above-mentioned embodiments can take 
the following aspects 

[0091] Information about a position or a locus on 
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which a working robot operates is marked on an object 
to be worked in advance. Then, a position of the marking 
on an image of a camera is detected by means tor spec- 
ifying points constituting a working line. 
[0092] The means for specifying points constituting a 
working line performs image processing for detecting a 
line section along the working line lo specify the points 
constituting the working line. 

[0093] The points constituting the working line speci- 
fied as well as the image of the camera are displayed 
on tho display moans. Then, when the point specified is 
different from a point to be recognized, an operator can 
indicate a correct point to be recognized on the camera 
image so that he/she can specify it again. 
[0094] When the points constituting the working line, 
which are to be recognized, are specified on the image 
of the camera, a feature amount required for detecting 
points to be recognized by image processing is recorded 
form the image of the camera. Then, the points consti- 
tuting the working line are specified by moving the 
teaching robot along the working path and performing 
image processing using the recorded feature amount 
(that is, in the next and subsequent recognition, the 
points constituting the working line are automatically 
specified by the image processing using the recorded 
feature amount). 

[0095] By performing an image processing on the ba- 
sis of a shape (a straight line, a circle, a curved line or 
the like) of a working line on the image of the camera 
specified by an operator in advance, points constituting 
the working line is specified. 

[0096] When the teaching robot is moved Irom the 
current image taking-in position to the next image tak- 
ing-ln position, the next image taking-in position is ob- 
tained on the basis of 3-dimensional positions of a plu- 
rality of points obtained by the current time. Also, the 
next image taking-in position is obtained on the basis of 
the 2-dimensional positions of the plurality of points con- 
stituting the working line on the image and the attitude 
of a plane, which have been specified until the current 
time. That is, the next image taking-in position is deter- 
mined using the 3-dimensional positions orthe 2-dimen- 
sional positions which have been specified until the cur- 
rent time, and the robot is moved to the image taking-in 
position. By moving the robot in this manner, the points 
constituting a working line are specified continuously. 
Also, when the next image taking-in position is obtained, 
the shape of the working line is considered. 
[0097] From the position of a measuring portion on the 
image of the camera and the position of the camera at 
the image taking-in time, a straight line (a view line of a 
camera) within a space connecting the measuring por- 
tion and the camera is obtained. The attitude of the cam- 
era or the projector is determined such that the meas- 
uring portion existing on the straight line can be meas- 
ured. The camera orthe projector is caused to approach 
to an object along the straight line so that it is moved up 
to a position at which 3-dimensional information can be 



measured. That is, prior to starting detection of points 
constituting the working line, from the position of a de- 
tection starting point on an image having a field of view 
wider than that at the recognition time ,a straight line 

* within a space in which the detection starting position 
exists is determined as a field of view of the camera, 
and then the 3-dimensional sensor (the projector or the 
camera) is caused to approach to the measuring posi- 
tion along the straight line. As a result, even when the 

io 3-dimensional position of the starting position of meas- 
urement is unknown, it becomes possible to cause the 
camera or the projector to approach to the position in a 
state where the starting point has been grasped correct- 
ly 

is [0098] The projecting means and the camera are at- 
tached to the robot at the time of teaching and they are 
detached therefrom at the time of working. 
[0099] The robot teaching apparatus is further provid- 
ed with means for creating a working program of the 

20 working robot from the 3-dimensional positions of the 
points constituting the working line specified. 



Claims 

25 

1 . A teaching apparatus which uses a teaching robot 
provided with projecting means for projecting a pat- 
tern light on an object, 

and a camera to obtain teaching points for a 
30 working robot for performing working along a work- 
ing line on the object, comprising: 

means for obtaining, from an Image of the cam- 
era about a face of the object on which the pat- 

35 tern light is projected, a 3 dimensional position 

and an attitude of the face; 
means for specifying points constituting the 
working line on the image of the camera; and 
means for obtaining 3-dimensional positions of 

40 the specified points constituting the working 

line on the basis of the 3-dimensional position 
and the attitude of the face, and the positions 
of the points constituting the working line on the 
image of the camera. 

45 

2. The teaching apparatus according to claim 1, 
wherein a single robot is used both as the working 
robot and the teaching robot. 

so 3. The teaching apparatus according to claim 1 or 2, 
wherein said specifying of the points constituting 
the working line is performed on an image of anoth- 
er camera mounted on another teaching robot or a 
moving device provided separately of the working 

55 robot, or on an image of another camera mounted 
in a fixing manner separately of the teaching robot. 

4. The teaching apparatus according to claim 1 , 2 or 
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3, further comprising: 



display means for displaying an image of the 
camera; and 

position pointing means for pointing a point on 
an image displayed on the display means, 

wherein said means for specifying points con- 
stituting the working line specifies a point on the im- 
age of the camera displayed on the display means 
according to a pointing of said position pointing 
moans. 

5. The teaching apparatus according to any preceding 
claim, wherein marking of information about a posi- 
tion or a locus where the working robot is to operate 
is performed on the object in advance, and said 
means for specifying points constituting the working 
line detects the position of the marking on the image 
of the camera. 20 

6. The teaching apparatus according to any preceding 
claim, wherein the means for specifying the points 
constituting the working line performs image 
processing for detecting a line secstiorualong^the^s... 
working line on an image of the camera to specify 
the points constituting the working line. 

7. The teaching apparatus according to claim 6, fur- 
ther comprising: 30 

display means for displaying the points consti- 
tuting the working line together with the image 
of the camera. 

35 

wherein, when the specified point is different 
from a point to be recognized, a point to be recog- 
nized on the image of the camera can be pointed 
by an operator for specifying the point again. 

40 

8. The teaching apparatus according to claim 6 or 7, 
wherein, when the point to be recognized is speci- 
fied on the image, a feature for detecting the point 
by image processing from the image is recorded, 
and 45 

the teaching robot moves along the working 
line and the teaching robot specifies the points con- 
stituting the working line by image processing from 
the recorded feature. 

so 

9. The teaching apparatus according to claim 6,7 or 8, 
wherein the image processing is performed on the 
working line on the image of the camera on the ba- 
sis of the shape of the working line specified by the 
operator so that specifying of the points constituting 
the working line is performed. 

10. The teaching apparatus according to any preceding 



claim, wherein, when the teaching robot moves 
from a current image taking-in position to the next 
image taking-in position, the next image taking-in 
position is obtained on the basts of 3-dimensional 
positions of a plurality of points which have been 
obtained until the current lime. 

1 1 . The teaching apparatus according to any of claims 
1 to 9, wherein, when the teaching robot moves 
from a current image taking-in position to the next 
image taking-in position, the next image taking-in 
position is obtained on the basis of 2-dimonsional 
positions of a plurality of points constituting the 
working tine which has been specified until the cur- 
rent time on the image, and the attitude of the face. 

12. The teaching apparatus according to claim 1 0 or 1 1 , 
wherein when the next image taking-in position is 
obtained, the next image taking-in position is ob- 
tained further on the basis of the shape of the work- 
ing line. 

13. The teaching apparatus according to any preceding 
claim, wherein, from a position of a measuring por- 

,._ tion on the image of the camera and a position of 
the camera at a time of the image taking-in. a 
straight line (a view line of the camera) within a 
space connecting the measuring portion and the 
camera is obtained, an attitude of the camera or the 
projector is determined such that the measuring 
portion existing on the straight line can be meas- 
ured, and the camera or the projecting means ap- 
proach to the object to be moved up to a position 
where 3-dimensional information can be measured. 

14. A teaching apparatus according to any preceding 
claim, wherein the projecting means and the cam- 
era are mounted on the teaching robot at a time of 
teaching and are detached therefrom at a time of 
working. 

15. A teaching apparatus according to any preceding 
claim, further comprising means for creating a work- 
ing program for the working robot from 3-dimen- 
sional positions of the specified point constituting 
the working line. 

16. A teaching apparatus according to any preceding 
claim, wherein said projecting means is operable to 
project said pattern light as a slit light or a spot light. 
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